Learning Nonlinear Overcomplete Representations for Eecient Coding

We derive a learning algorithm for inferring an overcomplete basis by viewing it as probabilistic model of the observed data. Over-complete bases allow for better approximation of the underlying statistical density. Using a Laplacian prior on the basis coeecients removes redundancy and leads to representations that are sparse and are a nonlinear function of the data. This can be viewed as a generalization of the technique of independent component analysis and provides a method for blind source separation of fewer mixtures than sources. We demonstrate the utility of overcom-plete representations on natural speech and show that compared to the traditional Fourier basis the inferred representations potentially have much greater coding eeciency. A traditional way to represent real-values signals is with Fourier or wavelet bases. A disadvantage of these bases, however, is that they are not specialized for any particular dataset. Principal component analysis (PCA) provides one means for nding an basis that is adapted for a dataset, but the basis vectors are restricted to be orthogonal. An extension of PCA called independent component analysis the learning of non-orthogonal bases. All of these bases are complete in the sense that they span the input space, but they are limited in terms of how well they can approximate the dataset's statistical density. Representations that are overcomplete, i.e. more basis vectors than input variables, can provide a better representation, because the basis vectors can be specialized for